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ABSTRACT 


Our  current  fiscal  environment  requires  serious  consideration  of  new  test  and  evaluation 
strategies  that  provide  an  accurate  assessment  of  program  risks.  With  this  is  in  mind,  is  it  truly 
possible  to  transition  from  the  conventional  test  and  evaluation  process  to  a  risk-based  model? 

Risk,  by  definition,  is  a  measure  of  future  uncertainties  in  achieving  program 
perfonnance  goals  and  objectives  within  defined  cost,  schedule,  and  performance  constraints. 
Risk  can  be  associated  with  all  aspects  of  a  program,  such  as  technology  maturity,  supplier 
capability,  design  maturation,  and  demonstrated  performance  against  planned  performance. 
Traditionally,  this  assessment  of  risk  and  the  development  of  mitigation  strategies  have  most 
often  resided  exclusively  within  the  program  manager’s  domain. 

The  purpose  of  this  Strategic  Research  Paper  is  to  determine  the  viability  of  a  risk-based 
approach  to  test  and  evaluation.  Research  is  based  on  a  limited  population  of  the  total  materiel 
acquisition  community  and  represents  an  Alpha  test  of  the  stated  issue.  If  proved  viable  with 
follow-on  research,  the  risk-based  approach  would  provide  senior  decision-makers  within  the 
Department  of  the  Army  with  an  independent  perspective  of  risks  based  on  test  data,  synthetic 
modeling  data,  and  other  relevant  analysis.  The  target  population  for  this  research  effort  will  be 
the  Anny  Test  and  Evaluation  Command  and  Program  Executive  Office  Intelligence,  Electronic 
Warfare  and  Sensors,  both  located  at  Aberdeen  Proving  Ground,  MD.  The  methods  that  will  be 
used  to  support  this  research  effort  will  be  a  quantitative  analysis  based  on  a  20-question  survey. 
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CHAPTER  1 
INTRODUCTION 


Background  and  Introduction 

The  Army  acquisition  community  is  in  the  midst  of  a  transformation  that  demands  the 
very  best  materiel  solutions  for  the  least  cost.  This  transfonnation  is  principally  driven  by 
changes  in  our  geopolitical  and  economic  environments.  While  all  serious  discussions  regarding 
the  future  of  programs  acknowledge  existing  capability  gaps,  senior  leaders  within  the  U.S. 
Anny  acquisition  community  are  under  constant  pressure  to  accurately  characterize  risks  and 
make  tough  decisions  regarding  the  future  course  of  their  acquisition  programs. 

In  his  statement  to  the  House  Anned  Services  Committee  on  January  26,  2011,  Defense 
Secretary  Robert  Gates  outlined  four  initiatives  that  the  Department  of  Defense  (DoD)  would 
take  to  reduce  spending.  He  offered  that  the  Services  would  achieve  $100  billion  in  efficiencies 
over  fiscal  year  (FY)  2012  through  FY  2016,  of  which  $29  billion  would  come  from  the  Anny. 
The  Services  would  also  develop  additional  efficiencies  to  accommodate  a  $78  billion  reduction 
in  the  top  line  to  support  the  administration’s  reduction  efforts.  Finally,  he  identified  significant 
personnel  reduction  as  a  means  to  achieve  additional  efficiencies  that  included  reductions  in  the 
contractor  workforce,  a  civilian  hiring  freeze,  reduced  civilian  workforce  levels  to  the  FY  2010 
level,  and  a  reduction  of  General  Officer  (GO)  and  Senior  Executive  Service  (SES)  billets  over 
the  next  2  years. 

There  is  no  doubt  that  our  current  fiscal  environment  demands  serious  consideration  of 
all  viable  test  and  evaluation  (T&E)  approaches  that  result  in  substantive  cost  savings  and  an 
accurate  assessment  of  program  risks.  With  this  in  mind,  is  it  possible  to  transition  from  a 
conventionally  based  construct  for  conducting  T&E  to  a  risk-based  approach?  The  viability  of 
such  a  transition  as  it  relates  to  fielding  complex  systems  in  times  of  constrained  resources  is 
examined  in  this  research.  Also  examined  are  the  viability  and  implications  of  moving  to  a  risk- 
based  approach  to  T&E.  Finally,  this  approach  may  provide  senior  decision-makers  with  an 
independent  perspective  of  risks  based  on  test  data,  synthetic  modeling  environments,  and 
relevant  analysis. 

Risk,  by  definition,  is  a  measure  of  future  uncertainties  in  achieving  program 
perfonnance  goals  and  objectives  within  defined  cost,  schedule,  and  perfonnance  constraints 
(Risk  Management  Guide  for  DoD  Acquisitions,  Sixth  Edition,  p.  1).  Risk  can  be  associated  with 
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all  aspects  of  a  program  such  as  technology  maturity,  supplier  capability,  design  maturation,  and 
demonstrated  performance  against  planned  performance.  Traditionally,  this  assessment  of  risk 
and  the  development  of  associated  mitigation  strategies  have  most  often  resided  exclusively 
within  the  program  manager’s  (PM’s)  domain. 

To  assist  in  managing  the  risks  involved  in  developing,  producing,  operating,  and 
sustaining  systems  and  capabilities,  the  acquisition  and  testing  communities  perform  T&E.  The 
PM,  in  concert  with  the  user  and  testers,  coordinates  developmental,  operational,  live-fire, 
system-of-systems,  interoperability,  information  assurance,  and  modeling  and  simulation 
activities  into  myriad  activities  commonly  referred  to  as  the  T&E  strategy.  This  strategy’s 
principal  objectives  are  to  provide  knowledge  that  will  assist  senior  leaders  in  managing  the 
technical  risks,  to  measure  progress  in  both  system  and  capability  development,  to  provide 
knowledge  of  system  capabilities  and  limitations  to  the  acquisition  community  for  use  in 
improving  system  performance,  and  to  provide  knowledge  to  the  user  community  for  optimizing 
system  use  in  operational  employment. 

The  U.S.  Army  Test  and  Evaluation  Command  (ATEC)  is  the  premier  organization 
chartered  with  the  responsibility  to  conduct  unbiased  T&E  for  the  U.S.  Anny.  In  addition  to 
planning,  executing,  and  reporting  test  and  evaluation  events,  it  also  serves  as  the  Independent 
Senior  Adviser  to  the  Secretary  of  the  Anny  and  the  Chief  of  Staff  of  the  Army  on  all  T&E 
matters. 

ATEC  recently  has  been  reorganized  to  optimize  efficiencies,  increase  effectiveness,  and 
provide  senior  leaders  with  the  absolute  best  information  and  recommendations  available  to 
support  senior  leader  decisions.  Under  the  new  reorganization,  the  command  consists  of  nine 
Developmental  Test  Centers  (DTCs),  Operational  Test  Command  (OTC),  and  Anny  Evaluation 
Center  (OTC). 

DTCs  represent  the  largest,  most  diverse  array  of  testing  capabilities  within  the  DoD. 
DTCs  test  military  hardware  of  every  description  under  precise  conditions  across  the  full 
spectrum  of  arctic,  tropic,  desert,  and  other  natural  or  controlled  environments  on  highly 
instrumented  ranges  and  test  courses.  These  centers  offer  a  full  range  of  test  services,  including 
providing  unbiased  test  data  on  the  technical  feasibility  of  early  concepts,  detennining  system 
perfonnance  and  safety,  assessing  technical  risks  during  system  development,  confirming 
designs,  and  validating  manufacturers’  facilities  and  processes  at  both  system  and  component 
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levels.  Their  testing  services  are  extended  to  all  DoD,  other  federal  agencies,  state  and  local 
governments,  foreign  and  allied  governments,  and  private  industry.  ATEC  accomplishes  its 
developmental  test  mission  at  eight  test  centers,  including  White  Sands  Test  Center  at  White 
Sand  Missile  Range,  NM;  West  Desert  Test  Center  at  Dugway  Proving  Ground,  UT;  Redstone 
Test  Center  at  Redstone  Arsenal,  AL;  Yuma  Test  Center  at  Yuma  Proving  Ground,  AZ; 
Aberdeen  Test  Center  at  Aberdeen  Proving  Ground  (APG),  MD;  Cold  Regions  Test  Center  at 
Fort  Greely,  AK;  and  the  Electronic  Proving  Ground  at  Fort  Huachuca,  AZ. 

As  the  U.S.  Anny  transitions  to  the  new  geopolitical  and  economic  environments,  so  does 
the  Anny’s  only  independent  operational  test  organization,  the  U.S.  Army  OTC  at  Fort  Hood, 
TX.  OTC  has  the  mission  to  conduct  realistic  operational  testing  in  the  critical  areas  of 
equipment,  doctrine,  force  design,  and  training.  Operational  tests  conducted  by  this  command  are 
required  by  public  law  and  provide  significant  data  to  Anny  decision-makers  on  key  systems  and 
concepts.  The  command  also  has  five  operational  test  directorates  at  critical  locations  throughout 
the  United  States.  OTC  uses  highly  sophisticated  and  state-of-the-art  technology,  both 
instrumentation  along  with  modeling  and  simulation,  to  collect  test  data  that  will  document 
perfonnance  levels  and  operational  suitability.  Perhaps  the  most  striking  initiative  by  the  Anny  is 
leveraging  the  capabilities  of  OTC  in  providing  an  operational  systems  of  systems  environment 
to  test  new  systems  for  operational  utility.  This  network-centric  test  mission  is  conducted  at 
White  Sands  Missile  Range,  NM,  and  represents  the  epicenter  of  T&E  for  the  Anny. 

This  network-centric  test  event,  known  as  Agile  Provider,  represents  the  largest 
operational  test  in  Anny  history.  It  is  a  cooperative  effort  between  ATEC,  Program  Executive 
Office-  Integration  (PEO-I),  and  the  Brigade  Modernization  Command  (BMC)  to  test  the  Army’s 
network.  It  also  represents  a  new  paradigm  for  operational  testing  that  includes  a  dedicated  test 
unit,  realistic  scenarios,  bundled  systems  testing  instead  of  a  one-system  test,  and  a  series  of 
network  integration  events  that  lead  to  a  capstone  test  in  fourth-quarter  FY  2012.  This  new 
paradigm  represents  significant  efficiencies  in  testing  and  will  save  more  than  130,000  man 
hours  alone  with  total  cost  reductions  of  more  than  $5.7  million  by  eliminating  duplication  of 
efforts  (Simmons,  2011). 

The  Anny  Evaluation  Center  (AEC)  is  the  organization  that  prepares  the  final  evaluation 
report  used  by  decision-makers  to  determine  whether  a  new  or  enhanced  system  will  become  part 
of  the  Army’s  arsenal.  In  this  role,  AEC  gets  involved  early  in  the  acquisition  process  to  ensure 
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that  T&E  programs,  strategies,  and  objectives  are  consistent  throughout  the  acquisition  program. 
Since  T&E  results  often  impact  decisions  reached  at  design  and  milestone  reviews,  early 
involvement  in  the  acquisition  process  adds  value  to  the  final  product  of  any  acquisition 
program.  AEC  evaluates  and  reports  on  each  system’s  effectiveness,  suitability,  and  survivability 
to  the  Anny’s  senior  leadership  and,  when  requested,  to  Congress.  AEC  organizes  and  employs 
ATEC  Systems  Teams  (AST)  for  each  system  under  evaluation  and  coordinates  input  into  the 
Test  and  Evaluation  Working  Integrated  Product  Teams  (T&E  WIPTs).  The  AST  Chair  serves  as 
the  lead  voice  and  ensures  that  the  T&E  strategy  is  designed  effectively  to  characterize  the 
perfonnance,  suitability,  and  survivability  of  the  system. 

A  New  Model  for  Risk-Based  Test  and  Evaluation 

U.S.  Army  Materiel  Systems  Analysis  Activity  (AMSAA)  is  an  analysis  organization  of 
the  Army  with  an  overall  goal  to  provide  soldiers  with  the  best  materiel  solutions  possible. 
AMSAA  supports  the  Anny  by  conducting  systems  and  engineering  analyses  to  support 
decisions  on  technology,  materiel  acquisitions,  and  the  designing,  developing,  and  sustaining  of 
Anny  weapon  systems.  In  its  presentation  to  the  2011  Anny  Operational  Research  Symposium, 
AMSAA  identified  a  technical  risk  methodology  that  included  a  13 -step  process  for  reducing 
technical  risks  across  the  materiel  development  community.  The  desired  end-state  of  the 
proposed  process  is  an  assessment  of  whether  the  proposed  technology  could  be  delivered  within 
the  required  timeframes  and  cost  targets.  This  assessment  risk  includes  the  development  effort, 
the  integration  effort,  and  the  manufacturing  effort  (Appendix  C).  The  central  premise  of  this 
research  paper  is  that  ATEC  is  well  positioned  to  marshal  its  resources  and  partner  with  AMSAA 
in  this  innovative  approach  to  risk-based  program  evaluations. 

The  process  would  begin  early  in  the  development  cycle  by  identifying  critical 
technologies  and  other  technologies  of  interest.  Each  technology  would  be  assessed  at  a 
particular  technology  readiness  level  (TRL).  Manufacturing  readiness  and  integration  readiness 
also  would  be  assessed  for  each  segment  of  technology.  Potential  risks  would  be  identified  and 
probability  distribution  would  be  prepared  for  each  technology.  These  distributions  would  be 
prepared  for  each  major  milestone  decision  within  the  life  cycle  of  the  program.  The  TRL, 
manufacturing  readiness  levels  (MRL),  and  integration  readiness  levels  (IRL)  would  be  assessed 
for  each  milestone  event  with  the  objective  readiness  as  follows:  TRL  7,  MRL  8,  and  IRL  8  by 
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Milestone  C.  This  new  risk-based  approach  to  T&E  would  arm  senior  leaders  with  innovative 
cost-sensitive  perspectives  as  they  conduct  key  milestone  decision  reviews. 

Overview  of  Methodology 

This  research  project  utilized  a  quantitative  research  methodology  involving  numerical 
representation  of  responses  received  from  the  target  population  described  below.  Data  were 
collected  to  test  the  hypothesis  and  investigate  the  perceptions  of  the  T&E  community  and  the 
program  management  community  with  regard  to  ATEC’s  active  role  in  risk  management.  The 
target  population  is  both  supervisory  and  nonsupervisory  personnel  from  AEC  and  the  Program 
Executive  Office  for  Intelligence,  Electronic  Warfare,  and  Sensors  (PEO  IEW&S).  PEO  IEW&S 
is  located  at  APG,  MD,  and  is  responsible  for  a  combination  of  more  than  110  programs  of 
record  and  quick  reaction  capabilities. 

Research  Questions 

This  strategic  research  paper  will  assess  the  potential  for  a  shift  in  focus  for  the  ATEC  in 
the  characterization  of  risks  for  the  program  management  community.  In  addition,  this  research 
paper  adds  to  the  volume  of  knowledge  on  potential  efficiencies  and  effectiveness  related  to  the 
formal  risk  management  processes  now  almost  exclusively  within  the  PM’s  domain.  This 
research  tests  the  hypothesis  that  the  Anny  would  benefit  from  a  shift  in  ATEC’s  mission  where 
the  principal  product  is  based  on  a  comprehensive  assessment  of  risk.  The  key  research  question 
supporting  this  hypothesis  is:  Should  the  methodology  for  evaluating  materiel  acquisition 
systems  within  the  U.S.  Anny  transition  from  the  delivery  of  traditional  T&E  to  a  risk-based 
assessment  model?  Other  supporting  research  questions  that  will  be  answered  as  a  result  of  this 
research  include:  Would  a  risk-based  assessment  prove  more  useful  to  PMs  and  Milestone 
Decision  Authorities  (MDAs)  than  the  current  method  of  T&E?  Should  a  risk-based  assessment 
apply  to  all  categories  of  materiel  acquisition  programs?  Would  an  independent  assessment  of 
the  cost,  schedule,  and  performance  risks  contribute  to  program  success? 

Research  Hypothesis 

AEC  leadership  supports  the  adaptation  of  evaluation  methodologies  that  accurately  and 
efficiently  characterize  risk  for  the  acquisition  community.  Likewise,  the  acquisition  community 
would  benefit  from  T&E  products  that  independently  characterize  program  risks  and  provide 
associated  mitigation  strategies  for  those  risk  areas.  This  leads  to  the  following  hypotheses: 
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1 .  The  current  evaluation  methodology  employed  by  AEC  should  be  changed  to  a  risk- 
focused  construct  that  directly  supports  the  PM’s  efforts  to  accurately  characterize 
program  risks. 

2.  The  current  evaluation  methodology  should  be  continued  by  AEC  and  should  not  be 
changed  to  a  risk-focused  construct  that  directly  supports  the  PM’s  efforts  to 
accurately  characterize  program  risk. 

Limitations  of  this  Study 

The  principal  limitation  of  this  study  relates  to  the  scope  and  magnitude  of  the  target 
population.  Although  the  Anny  has  numerous  PEOs  dispersed  throughout  various  locations 
within  the  continental  United  States,  it  is  beyond  the  scope  of  this  research  effort  to  effectively 
manage  the  survey  instrument  throughout  the  entire  materiel  acquisition  community.  Therefore, 
the  actual  target  population  for  this  research  effort  will  include  the  T&E  community  and  will  be 
limited  to  one  PEO  and  its  associated  program  management  offices  (PMOs). 
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CHAPTER  2 
LITERATURE  REVIEW 

Introduction  to  the  Literary  Review 

In  its  2003  Report  04-53  to  the  Senate  and  House  Anned  Services  Committees  on  revised 
acquisition  policy,  the  General  Accounting  Office — now  the  Government  Accountability  Office 
(GAO),  concluded  that  DoD  acquisition  policies  incorporate  best  practices  but  fail  to  implement 
sufficient  controls.  These  controls  apply  to  program  launch,  the  interval  between  integration  and 
demonstration,  and  at  production  commitment.  This  report  also  emphasizes  the  critical  point  that 
without  an  effective  risk  management  program  in  place,  aspiring  PMs  cannot  hope  to  sustain  the 
level  of  control  required  to  manage  a  successful  acquisition  program. 

While  the  main  catalyst  for  this  research  is  the  need  for  the  Army  to  effectively  leverage 
T&E  resources  in  the  area  of  risk  management,  there  is  a  wealth  of  scholarly  sources  available 
on  the  subject  of  risk.  As  a  result,  the  literature  search  begins  with  an  overview  of  DoD’s  and  the 
Army’s  risk  management  policies,  procedures,  and  objectives.  Academic  research  from 
commercial  sources  then  is  reviewed  to  ascertain  the  posture  of  risk  management  practices  with 
commercial  sectors.  Finally,  the  literary  review  will  progress  to  relevant  scholarly  presentation 
recently  presented  to  the  engineering  and  scientific  community  on  the  state  of  risk  management 
within  DoD  and  suggested  improvements. 

Department  of  Defense  Relevant  Literature 

The  principal  source  for  regulatory  guidance  within  DoD  relevant  to  risk  management  is 
Department  of  Defense  Directive  (DoDD)  5000.1,  The  Defense  Acquisition  System.  This 
document  provides  mandatory  provisions  that  the  materiel  acquisition  community  must  follow 
with  respect  to  all  aspects  of  program  management,  including  the  role  of  T&E.  It  provides 
specific  guidance  that  is  relevant  to  our  research — namely,  that  the  PM  must  balance  the  risks  of 
cost,  schedule,  and  perfonnance  to  keep  the  program  on  track  to  production  and  fielding.  It  is  the 
responsibility  of  decision-making  authorities  to  assess  risk  tradeoffs.  As  stated  in  DoDD  5000.1, 
The  Defense  Acquisition  System,  “Test  and  evaluation  shall  be  integrated  throughout  the  defense 
acquisition  process.  Test  and  evaluation  shall  be  structured  to  provide  essential  infonnation  to 
decision  makers,  assess  attainment  of  technical  performance  parameters,  and  determine  whether 
systems  are  operationally  effective,  suitable,  survivable,  and  safe  for  intended  use.  The  conduct 
of  test  and  evaluation,  integrated  with  modeling  and  simulation,  shall  facilitate  learning,  assess 
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technology  maturity  and  interoperability,  facilitate  integration  into  fielded  forces,  and  confirm 
performance  against  documented  capability  needs  and  adversary  capabilities  as  described  in  the 
system  threat  assessment.”  The  DoD  5000  series  clearly  sustains  the  notion  that  T&E 
organizations  should  play  an  integral  role  in  risk  management  for  DoD  materiel  acquisition 
systems. 

The  overarching  document  within  DoD  that  assists  PMs  in  effectively  managing  risks 
during  the  entire  materiel  acquisition  life  cycle  is  the  Risk  Management  Guide  for  DoD 
Acquisition,  Sixth  Edition.  This  commonly  used  source  acknowledges  that  the  purpose  of  risk 
management  is  to  help  ensure  program  cost,  schedule,  and  performance  objectives  are  achieved 
at  every  stage  in  the  life  cycle  and  to  communicate  to  all  stakeholders  the  process  of  managing 
program  uncertainties.  Although  this  guide  cannot  be  characterized  as  mandatory  guidance,  it 
provides  risk  management  elements  that  PMs  should  follow  as  they  apply  the  fundamental 
management  tools  provided.  One  of  the  more  notable  changes  to  this  document  is  the  emphasis 
given  tracking  the  status  of  risk  mitigation  implantation  vs.  risk  tracking  and  focusing  on  event- 
driven  technical  reviews  to  help  identify  risk  areas  and  ongoing  risk  mitigation  efforts.  These  are 
specific  areas  where  ATEC  could  potentially  leverage  its  considerable  T&E  expertise  in 
managing  program  risk. 

The  Defense  Acquisition  University’s  (DAU)  Test  and  Evaluation  Management  Guide  is 
another  relevant  product  that  reinforces  the  importance  of  substantive  test  and  evaluation 
involvement  in  the  risk  management  process.  It  is  important  to  note  that  the  T&E  process  is  an 
integral  part  of  the  Systems  Engineering  Process  (SEP),  which  identifies  levels  of  perfonnance 
and  assists  the  developer  in  correcting  deficiencies.  It  is  a  significant  element  in  the  decision¬ 
making  process,  providing  data  that  support  tradeoff  analysis,  risk  reduction,  and  requirements 
refinement.  Program  decisions  on  system  performance  maturity  and  readiness  to  advance  to  the 
next  phase  of  development  take  into  consideration  demonstrated  performance.  Finally,  the  T&E 
process  provides  data  that  advise  senior  leaders  within  the  Anny  how  well  the  system  is 
perfonning  during  development  and  if  it  is  ready  for  fielding. 

Academic  Research  from  Commercial  Sources 

Commercial  sources  and  academia  have  provided  a  wealth  of  information  related  to  risk 
management.  While  the  literary  search  revealed  much  of  this  research  was  related  to  specific 
niche  areas  within  the  commercial  sector,  the  following  sources  were  particularly  relevant.  In 
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their  paper,  A  new  approach  to  risk:  The  implications  of  E3,  Robert  Bea  and  other  members  of 
the  Civil  and  Environmental  Engineering  Department  at  the  University  of  California  at  Berkeley 
offered  some  particularly  useful  insights  into  the  effective  characterization  of  risk.  The  thesis  of 
this  paper  is  that  no  matter  how  much  physical  science  and  technology  are  involved  in  complex 
systems,  no  system  is  ever  purely  or  solely  physical  or  technical.  Certainly,  no  system  of  which 
we  are  aware  is  purely  scientific  or  technical  in  its  operation  or  management.  Furthermore,  while 
research  on  and  the  modeling  of  complex  systems  usually  rely  heavily  on  the  consideration  of 
technological  variables  and  processes,  they  typically  fail  to  consider  the  contributions  of 
individual  psychological,  organizational,  and  contextual  factors.  This  paper  argues  that  we  need 
more  effective  models  that  avoid  committing  errors  of  the  third  kind,  solving  the  wrong  problem. 
The  paper  concludes  that  risk  assessment  shapes  design,  construction  and  management  of 
infrastructure  systems  solutions,  so  great  attention  is  needed  on  how  it  is  done.  The  paper  is 
extremely  relevant  to  this  study  and  significantly  adds  to  the  body  of  research  on  risk 
management. 

Probabilistic  risk  assessment  concepts  provide  a  measure  of  statistical  rigor  when 
evaluating  program  risks.  Curtis  Smith  from  the  Idaho  National  Engineering  and  Environmental 
Laboratory  offers  useful  insights  into  this  area  of  study.  He  concludes  that  probabilistic  risk 
assessment  methodology  is  an  analysis  that  systematically  answers  the  questions:  What  can  go 
wrong,  how  likely  is  it  to  occur  (probability,  frequency),  and  what  will  be  the  outcome 
(consequences)?  The  SAPHIRE  software  tool  also  is  introduced  as  a  way  to  develop  those  risk 
concepts  dealing  with  event  trees,  fault  trees,  and  desired  end  states.  SAPHIRE  is  a  probabilistic 
risk,  and  reliability  assessment  software  tool.  SAPHIRE  stands  for  Systems  Analysis  Programs 
for  Hands-on  Integrated  Reliability  Evaluations  and  was  developed  for  the  U.S.  Nuclear 
Regulatory  Commission  (NRC)  by  the  Idaho  National  Laboratory.  Throughout  the  presentation, 
Curtis  Smith  emphasizes  the  importance  of  high-fidelity  data  analysis  that  quantifies  likelihood 
of  component  failure  and  ranks  individual  basic  events  relative  to  other  basic  events.  Finally,  he 
discusses  how  “importance  measures”  are  used  to  rank  events  and  are  vital  for  determining 
uncertainty.  These  “importance  measures”  are  comparable  to  the  “measures  of  effectiveness” 
employed  by  system  evaluators  as  they  assess  the  effectiveness,  suitability,  and  survivability  of 
new  programs  for  the  Army.  This  literary  source  also  adds  to  the  body  of  research  related  to  risk 
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management  and  provides  a  useful  construct  to  characterize  risk  in  objective  terms  that  provide  a 
measure  of  statistical  rigor. 

Relevant  Risk  Management  Scholarly  Presentations 

The  Army  Materiel  Analysis  has  contributed  much  to  this  specific  research  topic.  Three 
of  their  presentations  at  the  2011  Army  Operational  Research  Symposium  addressed  the  topic  of 
schedule,  cost,  and  performance  risk  methodologies.  These  presentations  provided  a  direct 
correlation  between  effective  program  management  and  an  innovative  approach  to  risk 
management. 

The  effective  measurement  of  technology  risk  with  each  materiel  solution  is  paramount  to 
program  success.  High  technology  risks  mean  technology  will  not  be  delivered  within  the 
required  timeframe  and  cost  target.  A  proposed  methodology  for  characterizing  technology  risk 
level  considers  the  likelihood  that  a  specific  technology  will  not  be  fully  developed,  integrated, 
and  manufactured  within  the  given  costs  and  schedule  constraints. 

In  considering  likelihood,  the  model  presents  three  variables  for  consideration.  The 
development  effort  itself  is  defined  as  the  level  of  effort  required  to  reach  TRL  7  by  Milestone  C. 
This  is  determined  by  considering  the  vendor’s  current  TRL  as  compared  to  the  probability  of 
reaching  TRL  7  by  Milestone  C  and  is  based  on  subject  matter  expertise  and/or  historical  data. 
The  next  variable  in  the  “Likelihood”  equation  was  the  integration  effort.  This  variable  considers 
the  current  level  of  integration  readiness  and  inversely  proportional  to  the  probability  to  reach 
IRL  7  by  Milestone  C.  The  final  variable  in  the  “Likelihood”  equation  is  the  manufacturing 
effort.  In  this  venue,  manufacturing  readiness  consist  of  an  assessment  of  manufacturing 
readiness  and  the  probability  to  reach  MRL  8  given  TRL  7  by  Milestone  C.  This  factor  also  is 
based  on  subject  matter  expertise  and/or  historical  data.  Finally,  a  Monte  Carlo  simulation  on 
probability  distributions  is  done  to  determine  expected  probability  and  is  defined  as  the 
likelihood  level  for  achieving  a  desired  technology  risk  level.  Data  for  these  variables  were 
obtained  through  a  12-step  process  known  as  technical  risk  assessment  approach.  The  process 
began  with  a  gathering  of  system  information  and  concludes  with  the  performance  of  a 
sensitivity  analysis  using  the  distribution  for  “Likelihood.” 

Effective  measurement  of  schedule  risk  also  is  an  issue  of  significance  to  the  program 
management  community.  A  schedule  risk  assessment  methodology  should  be  consistent, 
quantifiable,  and  repeatable.  While  the  immediate  schedule  risk  assessment  methodology  should 
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include  the  phase-driven  approach,  the  desired  end-state  for  this  methodology  is  clearly  to  evolve 
to  a  primarily  event-driven  approach.  The  proposed  methodology  would  include  hierarchical 
levels  that  include  schedules  from  the  management  level  (Level  I),  the  summary  schedule  (Level 
II),  the  control  schedule  (Level  III),  the  medium-term  schedule  (Level  IV),  and  the 
implementation  schedule  (Level  V).  Both  the  Management  Schedule  and  the  Control  Schedule 
are  required  for  the  Phased-Driven  approach  described  above.  This  approach  to  managing 
schedule  risk  consists  of  10  steps  that  begin  with  a  thorough  review  of  the  proposed  schedule  and 
culminates  with  the  generation  of  the  final  schedule  risk  analysis,  recommendations  to  mitigate 
schedule  drivers,  and  a  review  of  historical  lessons  learned.  Data  sources  for  this  methodology 
include  the  PM’s  Integrated  Master  Schedule  (IMS),  any  relevant  GAO  reports,  subject  matter 
expert  interviews,  relevant  historical  data,  as  well  as  other  sources  of  scheduling  data  that  may  be 
available  to  the  risk  management  team.  This  structured  approach  to  risk  management  is  expected 
to  deliver  consistent,  quantifiable,  and  repeatable  assessments  of  schedule  risk. 

The  final  and  perhaps  the  most  important  sources  in  the  literature  review  were  related  to 
the  assessment  of  cost  risk.  Few  would  argue  that  cost  risk  demands  effective  management  and 
the  complete  attention  of  the  PM.  This  is  especially  true  given  the  existing  budget  constraints 
associated  with  the  current  fiscal  environment.  While  this  element  of  risk  is  beyond  the  bounds 
of  traditional  T&E,  a  brief  overview  of  the  independent  risk  assessment  process  is  useful.  The 
Office  of  the  Deputy  Assistant  Secretary  of  the  Anny  for  Cost  and  Economics  (ODASA-CE) 
regularly  provides  overviews  of  cost  risk  and  uncertainty  analysis.  Without  an  effective  risk 
analysis,  a  cost  estimate  usually  will  be  a  point  estimate,  which  may  not  account  for  all  of  the 
uncertainties  inherent  in  the  program  effort.  Not  accounting  for  potential  uncertainties  may  lead 
to  underfunding,  cost  overruns,  and  the  potential  for  a  program  reduction  in  scope.  The  analysis 
methodology  begins  with  the  identification  of  areas  of  uncertainty.  The  uncertainty  then  is 
addressed  within  the  current  program  cost  estimate,  after  which  the  risk  is  quantified  within  the 
cost  estimate,  and  the  results  are  presented  to  relevant  stakeholders.  At  the  conclusion  of  the  risk 
management  process  and  in  preparation  for  the  MDA  briefing,  the  technical,  schedule,  and  cost 
risk  assessments  will  be  summarized  to  identify  key  risk  drivers,  to  show  any  interactions  and 
interdependencies  between  risk  categories,  and  the  presentation  of  risk  mitigation  strategies 
and/or  potential  tradeoff  for  each  alternatives. 
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Summary 

Developing  effective  risk  management  methodologies  provides  challenges  for  the  entire 
acquisition  workforce  as  it  performs  critical  functions  with  constrained  resources.  Whether  the 
risk  is  technology  based,  schedule  based,  or  cost  based,  it  is  imperative  that  we  provide  develop 
accurate  and  repeatable  assessment  to  senior  leaders.  Given  the  limited  resources  available  to  the 
PM,  we  cannot  afford  to  rely  on  marginal  assessments  that  support  critical  acquisition  decisions. 

This  literature  review  provides  a  glimpse  of  the  voluminous  studies,  reviews,  and  articles 
pertaining  to  risk  management.  It  also  illustrates  the  currently  constrained  environment  that 
acquisition  professionals  operate  in  as  they  deliver  quality  materiel  solutions  to  the  Anny. 
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CHAPTER  3 

RESEARCH  METHODOLOGY 

Introduction 

The  applied  research  methodology  supporting  this  strategic  research  paper  conformed  to 
the  descriptive  category.  Research  was  used  to  describe  facts  and  characteristics  of  a  given 
population  systematically,  factually,  and  accurately. 

Data  were  collected  in  order  to  test  the  hypothesis  using  quantitative  methods:  The  risk- 
based  test  and  evaluation  concept,  originally  developed  by  AMSAA  and  further  developed  in  this 
paper,  provides  a  useful  construct  in  assessing  program  risk.  Data  were  collected  through  the  use 
of  a  survey  instrument  administered  to  and  completed  by  members  of  the  T&E  community  as 
well  as  the  program  management  community.  The  subsequent  analysis  from  the  survey 
instrument  relies  on  quantitative  analysis  to  accurately  characterize  inputs  from  various  members 
of  the  materiel  acquisition  community. 

Research  Perspective 

The  research  perspective  for  this  study  is  derived  from  that  of  the  Anny  T&E  community 
as  well  as  a  representative  sample  of  the  program  management  community  and  their  viewpoint 
on  the  risk-based  approach  to  T&E.  This  target  population  consists  of  40  senior  leaders  from  the 
Intelligence,  Electronic  Warfare,  and  Sensors  program  management  community  at  APG,  MD. 
This  research  seeks  to  assess  the  benefit  and  usefulness  of  a  transition  from  traditional  T&E  to  a 
risk-based  approach.  Surveys  were  administered  anonymously  through  Army  Knowledge  Online 
(AKO)  using  SurveyMonkey  (surveymonkey.com)  as  the  principal  data  interface  mechanism. 
Tier  I,  Tier  II,  and  other  senior  technical  acquisition  personnel  completed  the  survey.  While  the 
scope  of  this  study  is  purposely  narrow,  the  views  and  experiences  of  this  segment  of  the  total 
population  can  be  reasonably  expected  to  be  similar  to  that  of  other  commodity  areas  that  were 
not  part  of  this  study. 

Research  Design 

The  principal  method  chosen  for  this  project  included  use  of  a  survey  instrument.  The  use 
of  this  quantitative  method  addressed  the  basic  research  question:  Should  the  methodology  for 
evaluation  materiel  acquisition  systems  within  the  U.S.  Army  transition  from  traditional  T&E  to 
a  risk-based  assessment  model?  The  survey  data  were  received  and  managed  by  a  commercial, 
online  product  that  provided  cross  tabulation  capabilities.  The  sampling  approach  applied  for  this 
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research  consisted  of  representative  program  management  organizations  within  a  single  specific 
commodity  area. 

Research  Instruments 

The  following  survey  instrument  was  utilized  to  collect  quantitative  inputs  for  this 
research  paper.  The  survey  consisted  of  20  questions  that  included  both  demographical  as  well  as 
substantive  questions  related  to  risk  management.  Additionally,  questions  were  targeted  to  assess 
respective  levels  of  usefulness,  overall  satisfaction  with  current  T&E  products,  and 
characterization  of  existing  risk  processes. 

Participants,  Population,  and  Sample 

The  survey’s  target  population  was  senior  leaders  from  the  PEO  IEW&S  acquisition 
workforce.  Participants  included  senior  military  officers,  nonsupervisory  (GS-13  to  GS-14), 
supervisory  (GS-14  to  GS-15)  and  SES  individuals.  These  employees  are  in  the  T&E  Career 
field  and  the  Program  Management  Career  field  at  APG,  MD.  Although  PEO  IEW&S  has  a 
workforce  of  approximately  600,  the  actual  population  of  PMs,  deputy  PMs,  product  managers, 
deputy  product  managers,  and  senior  T&E  personnel  is  approximately  40.  These  personnel 
represent  the  target  population  for  this  survey. 

Methodology  Summary 

This  research  was  designed  to  detennine  if  a  migration  to  a  risk-based  approach  to  T&E 
would  be  of  interest  to  selected  members  of  the  acquisition  community  on  APG,  MD.  As  a  first 
step,  this  research  has  produced  data  that  afford  the  reader  an  opportunity  to  better  understand 
the  complexities  associated  with  effective  risk  management  as  well  as  the  potential  usefulness  of 
a  new  risk  construct.  As  with  all  survey  instruments,  there  is  a  known  bias  in  the  collection  of 
opinion  data,  and  there  are  questions  regarding  the  complete  accuracy  and  validity  of  the 
employee  population  data.  Nonetheless,  the  methodology  is  believed  sufficient  to  allow 
representative  acquisition  workforce  members  to  make  an  assessment  regarding  the  extent  and 
potential  usefulness  of  the  risk-based  approach  to  T&E. 
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CHAPTER  4 

DATA  ANALYSIS  AND  RESULTS 


Methodology  Summary 

This  chapter  presents  the  results  of  the  research.  The  first  section  describes  the  target 
population  and  provides  relevant  demographics  such  as  number  of  years  of  experience,  current 
grade,  program  management  experience,  and  current  level  of  responsibility.  The  second  section 
provides  response  data  based  on  the  use  of  this  survey  instrument.  The  actual  survey  instrument 
is  provided  as  an  attachment  to  this  research  paper  at  Appendix  A. 

Population,  Sample,  and  Participants 

PEO  IEW&S  is  made  up  of  a  team  of  seven  project  managers/directors  dedicated  to 
delivering  designing,  delivering,  and  sustaining  advanced  technologies.  By  providing  systems 
that  cover  the  full  spectrum  of  warfighters’  needs,  the  PEO  impacts  virtually  every  mission  from 
providing  a  persistent  view  of  the  battlefield  to  protecting  aircrews  from  enemy  threats.  PEO 
IEW&S  develops  and  integrates  sensors  and  sensor  data  across  multiple  technologies,  ensuring 
warfighters  have  a  complete  understanding  of  the  battlefield.  This  is  achieved  through  the 
assimilation  of  sensor  infonnation  into  relevant,  timely  products  that  can  be  used  for  targeting, 
situational  awareness,  force  protection,  and  Reconnaissance,  Surveillance,  and  Target 
Acquisition  (RSTA).  PEO  IEW&S  has  lead  roles  in  most  critical  current  operations  efforts  to 
include  Counter  Improvised  Explosive  Devices  (IEDs),  sensors  for  Counter  Rocket  and  Mortar, 
the  ISR  Surge,  and  the  Integrated  Intelligence  Architecture  (I2A).  The  PEO  also  is  responsible 
for  a  combination  of  more  than  110  programs  of  record  and  quick  reaction  capabilities.  Fielded 
systems  are  utilized  for  a  large  array  of  missions  from  countering  IEDs,  to  aircraft  survivability. 
By  providing  a  bevy  of  sensors,  radars,  intelligence  collection  and  dissemination  equipment,  our 
most  important  soldiers  are  given  a  complete  picture  of  the  battle  space. 

The  target  population  for  this  survey  consisted  of  Tier  I  and  Tier  II  level  personnel  within 
PEO  IEW&S  along  with  associated  PMOs.  Of  the  40  personnel  affiliated  with  this  population, 
24  responses  were  received,  resulting  in  a  60  percent  response  rate. 

While  all  participants  were  members  of  the  Anny  Acquisition  Corps,  they  were  asked  to 
identify  their  assigned  acquisition  specialty  which  role  they  fulfill  within  the  acquisition 
community.  Results  are  shown  in  Figure  1 . 
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Figure  1.  Assigned  Specialty  Areas 

There  were  four  specialty  areas  identified  by  this  target  population.  More  than  7 1  percent 
of  the  total  respondents  indicated  they  were  assigned  in  the  program  management  specialty,  with 
the  remaining  personnel  reporting  T&E  and  science  and  technology  specialties.  These 
demographics  substantiate  the  intent  of  shaping  a  target  population  consisting  of  senior  program 
management  personnel.  Respondents  also  were  asked  to  identify  their  levels  of  experience.  Fifty- 
five  percent  of  respondents  were  experienced  professionals  with  more  than  20  years  of 
experience  in  acquisition-related  positions.  Forty-six  percent  of  respondents  reported  varying 
levels  of  acquisition  experience  ranging  from  6  to  20  years.  This  demographic  confirms  the 
earlier  statement  that  the  target  population  consists  of  senior  professionals  with  substantive 
experience  in  materiel  acquisition. 
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How  many  years  of  experience  do  you  have  in  an  acquisition  related  position? 


Six  to  ten  years  Fifteen  to  twenty  years 

Less  than  fKre  years  Eleven  to  fifteen  years  More  than  twenty  years 


Figure  2.  Years  of  Experience 


The  survey  instrument  also  asked  respondents  to  identify  their  employment  status  as 
indicated  in  Figure  3.  The  reported  demographic  indicated  more  than  70  percent  Department  of 
the  Army  (DA)  civilians  and  2 1  percent  military.  The  remaining  8  percent  were  contractors  who 
provided  direct  support  to  senior  acquisition  officials. 
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What  is  your  current  employment  status? 


Figure  3.  Employment  Status 

By  far,  the  largest  demographic  in  the  population  were  DA  Civilians  at  the  grades  of  GS- 
14  and  GS-15,  with  more  than  62  percent.  Military  officers  represented  the  next  largest 
demographic,  with  17  percent.  The  survey  respondents  also  included  one  GO  and  one  member  of 
the  SES. 
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What  is  your  current  rank/grade  level? 
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Figure  4.  Current  Grade 

Seventy-five  percent  of  respondents  indicated  they  had  experience  working  large 
complex  acquisition  programs  designated  as  acquisition  category  (ACAT)  I.  The  remaining 
respondents  reported  experience  ACAT  II  and  III  as  their  largest  programs. 

Survey  Results 

Respondents  also  were  asked  to  assess  the  usefulness  of  ATEC  products  at  varying  stages 
of  the  materiel  acquisition  life  cycle.  The  largest  segment  of  the  population  that  reported  the 
highest  levels  of  usefulness  for  ATEC  products  were  those  respondents  preparing  for  a  Milestone 
C  or  Full  Rate  Production  decision.  The  largest  segments  of  the  population  reporting  a  minimal 
value  of  satisfaction  were  those  preparing  for  Program  Initiation  or  a  Milestone  B  decision. 
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At  which  major  decision  points  do  you  find  Army  test  and  evaluation 
products  most  useful? 


Prior  to  Program 
Initiation 


Milestone  A 


Milestone  B 


Milestone  C 


Full  Rate  Production 


Materiel  Release 


Type  Classification 


I  Not  Very  Useful 
I  Somewhat  Useful 
I  Satisfied 
I  Very  Useful 
I  Extremely  Useful 


Figure  5.  Usefulness  of  ATEC  Products 

A  cross-tabulation  of  this  question  also  was  done  to  compare  levels  of  satisfaction 
between  senior  and  more  junior  personnel.  While  all  grades  reported  varying  levels  of 
satisfaction  with  T&E  products,  clearly  the  GS-14/15  segment  reported  the  highest  levels  of 
satisfaction. 
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Figure  6.  Levels  of  Satisfaction 


A  cross-tabulation  of  this  question  also  was  done  to  compare  levels  of  satisfaction 
between  personnel  who  identified  themselves  as  Producers  and  Recipients  of  T&E  products. 
While  both  segments  reported  varying  levels  of  satisfaction  with  T&E  products,  clearly  the 
producers  reported  highest  levels  of  satisfaction. 

Respondents  were  asked  to  identify  the  highest  level  of  program  they  had  been  affiliated 
with.  Given  the  complex  nature  of  major  programs  and  increasing  oversight  from  the  Office  of 
the  Secretary  of  Defense  (OSD),  external  audit  agencies  and  GAO,  responses  to  this  item 
provided  a  contextual  framework  on  whether  a  T&E  based  risk  management  approach  would  be 
useful.  Of  the  total  responses,  75  percent  reported  AC  AT  1  programs  as  their  highest  level,  with 
the  remaining  25  percent  reporting  ACAT  II/III.  These  data  may  also  be  characteristic  of  the 
specific  functional  areas  associated  with  the  target  organization. 
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Figure  7.  ACAT  Levels 


Respondents  were  asked  to  assess  the  overall  usefulness  of  government  T&E  products 
compared  to  commercial  products.  A  significant  majority,  61  percent,  of  respondents  believed 
that  data  from  multiple  commercial  and  government  sources  were  more  useful  than  a  single¬ 
source  approach  to  evaluation.  Thirty- five  percent  felt  that  government  T&E  reports  were  most 
useful,  while  4  percent  believed  contractor  reports  were  sufficient  to  assess  the  effectiveness, 
suitability,  and  survivability  of  major  system  acquisitions. 
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Which  of  the  following  test  and  evaluation  sources  do  you  find 

most  useful? 

60.9  % 


Contractor  Test  Reports 

Army  Test  and 
Evaluation  Reports 

Both  are  equally  useful 


4.3  % 

34.8  % 


Figure  8.  Usefulness  of  Government  vs.  Commercial  T&E  Products 

Respondents  were  asked  to  assess  the  overall  quality  of  government  T&E  products 
derived  from  various  sources.  Possible  responses  ranged  from  “Not  Very  Satisfied”  to 
“Extremely  Satisfied.”  Possible  products  included  developmental  test  reports,  operational  test 
reports,  fonnal  system  evaluation  reports,  abbreviated  capabilities  and  limitation  reports,  and 
finally,  forward  areas  assessment  reports.  Of  the  total  respondents  who  indicated  an  extremely 
satisfied  level  of  satisfaction,  developmental  tests  reports  received  the  highest  level  of 
satisfaction  with  32  percent,  followed  by  operational  test  reports  with  21  percent.  The  formal 
evaluation  reports  received  extreme  satisfaction  levels  of  17  percent,  while  the  Capabilities  and 
Limitation  Reports  and  the  Forward  Area  Assessment  Reports  both  received  extreme  satisfaction 
levels  of  15  percent  each.  Those  respondents  unsatisfied  with  the  overall  quality  of  T&E 
products  were  the  smallest  group  of  respondents. 
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How  satisfied  are  you  with  the  overall  quality  of  the  following  Army 
test  and  evaluation  products? 


Operational  Test  Reports  Capabilities  and 

Limitations  Reports 


Developmental  Evaluation  Reports  Forward  Area 

Test  Reports  Assessment  Reports 


I  Not  Very  Satisfied 
l  Somewhat  Satisfied 
I  Satisfied 
I  Very  Satisfied 
I  Extremely  Satisfied 


Figure  9.  Levels  of  Satisfaction  with  T&E  Products 

Respondents  were  asked  to  assess  the  overall  timeliness  of  government  T&E  products 
derived  from  various  sources.  As  with  the  previous  question,  possible  responses  ranged  from 
“Not  Very  Satisfied”  to  “Extremely  Satisfied.”  Possible  products  included  developmental  test 
reports,  operational  test  reports,  fonnal  system  evaluation  reports,  abbreviated  capabilities  and 
limitation  reports,  and  finally,  forward  areas  assessment  reports.  Of  the  total  respondents 
indicating  they  were  extremely  satisfied  about  timeliness,  developmental  test  reports  received  the 
highest  level  of  satisfaction  with  28  percent.  Twenty-eight  percent  of  respondents  believed  there 
were  timeliness  issues  with  formal  evaluation  reports. 
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How  satisfied  are  you  with  the  overall  timeliness  of  the  following 
Army  test  and  evaluation  products? 


Operational  Test  Reports  Capabilities  and 

Limitations  Report 


Developmental  Evaluation  Reports  Forward  Area 

Test  Reports  Assessment  Reports 


I  Not  very  satisfied 
l  Somewhat  satisfied 
I  Satisfied 
I  Very  Satisfied 
I  Extremely  Satisfied 


Figure  10.  Timeliness  of  T&E  Products 

Respondents  were  asked  how  often  they  conducted  Risk  Review  Boards  within  their 
respective  PMOs.  The  survey  data  revealed  that  risk  management  provided  significant  program 
metrics  and  commanded  significant  meeting  time.  Thirty-eight  percent  reported  having  monthly 
review  boards  and  33  percent  reported  risk  review  boards  prior  to  major  milestone  decisions. 
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How  often  does  your  group  hold  Risk  Review  Board  meetings? 


37.5  % 


33.3  7. 


Figure  11.  Risk  Review  Boards 

Respondents  were  asked  whether  their  PMOs  maintained  Risk  Management  Plans  as  a 
formal  mechanism  to  manage  risks.  A  full  83  percent  reported  having  a  written  risk  management 
plan,  with  only  9  percent  reporting  not  having  a  risk  management  plan.  This  small  percentage 
may  reflect  personnel  who  are  not  assigned  to  specific  PMOs  but  rather  work  in  a  staff  position. 
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Does  your  organization  have  a  written  Risk  Management  Plan? 


Figure  12.  Risk  Management  Plans 

Respondents  were  asked  to  identify  those  steps  of  the  risk  management  process  that 
would  benefit  from  a  risk-based  approach  to  T&E.  The  vast  majority  of  respondents  perceived 
that  there  would  be  some  benefit,  some  moderate  benefit,  or  substantial  benefit  to  the  various 
steps  of  their  risk  management  program  through  the  use  of  risk-based  T&E  products.  As  shown 
in  Figure  14,  only  a  small  percentage  believed  that  risk-based  T&E  products  would  not  have 
provided  added  value  to  the  risk  management  process. 
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Members  of  the  target  population  were  asked  if  they  believed  ATEC  should  expand  the 
focus  of  its  analysis  to  include  an  assessment  of  cost  risk.  In  particular,  the  question  focused  the 
respondent  on  the  difference  between  “should  costs”  and  “will  costs”  and  included  cost  risks 
associated  with  the  overall  program  as  well  as  costs  associated  with  the  program’s  T&E  events. 
Thirty-nine  percent  of  respondents  perceived  there  to  be  no  value  in  ATEC’s  involvement  in 
program  cost  analysis,  however,  61  percent  of  respondents  perceived  there  to  be  varying  levels 
of  value  from  ATEC’s  involvement  in  analysis.  Levels  of  value  ranged  from  “somewhat  useful” 
to  “very  useful”  to  “extremely  useful.”  A  significant  majority  of  respondents  believed  ATEC 
should  be  involved  in  analysis  of  costs  associated  with  the  planning,  execution,  and  reporting  of 
test  events. 
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Would  it  be  helpful  if  Army  test  and  evaluation  products  conducted 
cost  based  risk  analysis  to  the  following  areas? 


Costs  (Program  Level 
Should  Cost  vs.  Will  Cost) 


Costs  (Test  and 
Evaluation  Should 
Cost  vs.  Will  Cost) 


Not  Very  Useful 
IMI  Somewhat  Useful 
Very  Useful 
Hi  Extremely  Useful 


Figure  14.  Cost-Based  Risk  Assessment 

Members  of  the  target  population  were  asked  if  they  believed  ATEC  should  expand  the 
focus  of  its  analysis  to  include  an  assessment  of  schedule  risk.  This  question  was  framed  to 
include  risk  associated  with  the  program’s  IMS  and  the  specific  schedule  associated  with  T&E 
events.  Sixty-five  percent  of  respondents  perceived  there  would  be  varying  degrees  of  value 
associated  with  an  external  assessment  of  schedule  risk  from  the  T&E  community,  while  only  35 
percent  rendered  a  perception  of  minimal  value.  With  regard  to  those  risks  associated  with  the 
T&E  schedule,  83  percent  perceived  varying  degrees  of  usefulness  with  an  external  assessment 
of  schedule  risks. 
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Would  it  be  helpful  if  Army  test  and  evaluation  products  conducted 
schedule  based  risk  analysis  to  the  following  areas? 


Schedule  (Integrated  Schedule  (Test  and 

Master  Schedule)  Evaluation  Schedules) 


Figure  15.  Schedule  Risk  Assessment 

Respondents  were  asked  if  it  would  be  helpful  if  Army  T&E  products  conducted 
performance-based  risk  analysis  in  the  areas  of  effectiveness,  suitability,  and  survivability. 
Eighty-four  percent  believed  it  would  help  in  assessing  effectiveness  risks,  83  percent  believed  it 
would  help  in  assessing  suitability  risks,  and  83  percent  believed  that  a  risk-based  approach  to 
T&E  would  help  in  assessing  survivability  risks. 
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Would  it  be  helpful  if  Army  test  and  evaluation  products  conducted 
performance  based  risk  analysis  to  the  following  areas? 


System  Effectiveness  System  Suitability  System  Survivability 


I  Not  very  useful 
Useful 

I  Very  Useful 
I  Extremely  Useful 


Figure  16.  Performance  Risks 


Respondents  were  asked  “Who  should  be  the  principle  recipient  of  risk-based  T&E 
products?”  Forty-two  percent  of  respondents  believed  the  PM  should  be  the  principle  recipient  of 
T&E  products.  Forty-two  percent  also  believed  it  was  important  for  all  stakeholders  to  be 
principal  recipients  of  T&E  products.  Only  4  percent,  8  percent,  and  4  percent  believed  the 
Combat  Developer,  PEO,  or  the  MDA  should  be  the  principal  recipient  of  risk-based  T&E 
products. 
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Who  should  be  the  principle  recipient  of  risk-based  test  and 
evaluation  products? 


TRADOC  Capabilities 
Manager 

Program  Manager 

Program  Executive 
Officer  (ACAT I  and  II) 

Milestone  Decision 
Authority 

All  of  the  above 


41.7% 

Figure  17.  Recipients  of  Risk-Based  T&E  Products 

Participant  Comments 

Participants  were  given  the  opportunity  to  provide  comments  on  a  Risk-Based  T&E 
approach.  Comments  varied.  However,  60  percent  of  the  comments  indicated  a  preference  for  a 
shift  to  risk-based  T&E,  15  percent  of  the  comments  reflected  a  resistance  to  shifting  to  a  risk- 
based  approach  to  T&E,  and  25  percent  contained  other  nonrelated  comments.  A  complete 
listing  of  all  comments  is  found  at  Appendix  B. 

Summary  of  Results 

The  target  population  for  this  research  consisted  of  selected  senior  leaders  directly 
engaged  in  materiel  acquisition  activity  and  responsible  for  evaluating  programmatic  risks.  To 
create  proper  boundaries  for  this  research  effort,  the  target  population  was  limited  to  one  PEO 
and  its  respective  PMs  and  leaders  associated  with  Intelligence,  Electronic  Warfare,  and  Sensors 
systems.  Of  the  40  affiliated  workforce  members,  24  responses  were  received,  resulting  in  a  60 
percent  response  rate. 
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The  survey  was  designed  to  collect  information  regarding  the  respondents’  personal 
characteristics,  their  professional  experience,  and  their  perspectives  on  the  value  of  a  risk-based 
approach  to  T&E.  Survey  results  indicated  that  the  vast  majority  of  respondents  had  substantive 
acquisition  experience  with  major  programs  of  record  that  required  effective  employment  of  a 
risk  management  program.  Response  data  were  then  analyzed  to  determine  if  the  members 
perceived  the  positive  benefits  of  a  risk-based  approach  to  T&E.  While  responses  varied  with 
respect  to  the  evaluation  of  schedule  and  cost  risks,  the  vast  majority  of  respondents  believed  that 
a  risk-based  approach  to  T&E  would  add  value  to  the  risk  management  process,  especially  in 
evaluating  those  risks  associated  with  the  effectiveness,  suitability,  and  survivability  of  major 
programs  of  record. 
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CHAPTER  5 

CONCLUSIONS  AND  RECOMMENDATIONS 


Conclusions 

The  intent  of  this  strategic  research  paper  was  to  detennine  the  viability  of  a  risk-based 
approach  to  T&E  for  the  materiel  acquisition  community  at  large.  In  doing  so,  the  research  effort 
considered  the  perceptions  of  senior  leadership  from  representative  program  management 
organizations  regarding  the  value  of  T&E  products  within  the  construct  of  risk  management. 
This  chapter  of  the  strategic  research  paper  provides  an  overview  of  the  research  results, 
identifies  the  limitations  of  this  research,  draws  conclusions,  and  provides  recommendations 
regarding  the  implications  for  further  research. 

Risk  is  a  measure  of  future  uncertainties  in  achieving  program  perfonnance  goals  and 
objectives  within  defined  cost,  schedule,  and  performance  constraints.  While  this  measurement 
has  traditionally  resided  exclusively  within  the  PM’s  domain,  this  research  has  shown  it  would 
add  value  to  have  additional  risk  management  input  from  external  sources.  Though  there  is  some 
evidence  PMs  also  would  benefit  from  external  schedule  and  cost  risk  analysis,  this  research 
substantiates  the  value  of  the  T&E  community  in  assessing  system  perfonnance.  This  research 
has  clearly  shown  that  the  program  management  community  values  the  early  involvement  of  the 
test  community  as  it  relates  to  effectiveness,  suitability,  and  survivability  of  system  performance. 

This  research  has  also  shown  that  senior  leaders  perceive  the  value  of  a  shift  in  paradigms 
from  the  existing  construct  presently  used  by  testers  to  a  risk-based  model.  They  believe  this 
model  should  be  applied  throughout  the  life  cycle  of  the  program  to  achieve  maximum  benefit. 
Respondents  indicated  the  process  should  begin  early  in  the  development  cycle  by  identifying 
critical  technologies  and  other  technologies  of  interest.  Each  technology  should  be  assessed  at  a 
particular  technology  readiness  level.  Furthennore,  manufacturing  readiness  and  integration 
readiness  also  should  be  assessed  for  each  segment  of  technology.  Potential  cost,  schedule,  and 
perfonnance  risks  should  be  identified,  and  probability  distribution  would  be  prepared  for  each 
technology.  These  distributions  should  be  prepared  for  each  major  milestone  decision  event 
within  the  life  cycle  of  the  program.  This  new  risk-based  approach  to  T&E  would  sufficiently 
ann  senior  leaders  with  innovative  cost-sensitive  perspectives  as  they  conduct  key  milestone 
decision  reviews. 
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Given  the  complex  nature  of  major  programs  and  increasing  oversight  from  OSD, 
external  audit  agencies  and  GAO,  this  risk-based  approach  would  benefit  PMs  as  they  manage 
key  stakeholders  and  risks  associated  with  their  programs.  This  research  also  indicated  a 
significant  majority  of  senior  leaders  believed  that  the  PM  should  be  the  principal  recipient  of 
risk-based  T&E  products.  Although  other  key  stakeholders  also  should  be  integrated  into  the 
overall  risk  coordination  community,  the  PM  is  ultimately  responsible  for  the  identification, 
characterization,  and  mitigation  of  program  risks. 

Another  significant  finding  associated  with  this  research  is  the  value  that  PMs  place  on 
multiple  sources  of  risk  management  information.  Senior  leaders  consistently  reported  the  use  of 
a  formal  risk  management  plan  and  systemic  risk  review  panels  to  synchronize  program  risks. 
Data  from  multiple  government  and  commercial  sources  must  be  verified,  validated,  and 
synchronized  into  the  overall  risk  management  package.  While  “more”  is  not  necessarily  better, 
the  test  community  is  well  equipped  to  assist  in  this  regard. 

Finally,  this  research  proved  that  members  of  the  program  management  community  are 
indeed  satisfied  with  the  overall  quality  of  government  T&E  products  derived  from  various 
sources.  These  sources  include  developmental  test  reports,  operational  test  reports,  fonnal  system 
evaluation  reports,  abbreviated  capabilities  and  limitation  reports,  and  forward  area  assessment 
reports.  Sustaining  this  exceptional  level  of  credibility  will  remain  of  utmost  importance  as  the 
test  community  migrates  to  a  new  risk-based  model  for  test  and  evaluation. 

Strengths  and  Limitations  of  this  Study 

A  key  limitation  of  this  research  is  that  it  purposely  focused  on  a  very  limited  number  of 
organizations  and  senior  leaders  at  APG.  The  target  population  consisted  of  40  senior  leaders 
from  one  PEO  and  its  associated  PMs.  While  this  research  provided  adequate  coverage  for  this 
segment  of  the  acquisition  community,  it  should  be  expanded  to  include  a  broader  segment  of  the 
materiel  acquisition  community. 

Another  key  limitation  is  the  low  overall  response  rate  to  the  survey.  Of  the  40  possible 
respondents,  only  24  actually  completed  and  returned  the  survey  instrument  as  requested, 
resulting  in  a  60  percent  response  rate.  Even  though  the  response  rate  was  relatively  low,  the  data 
provide  good  insight  into  senior  leader  views  on  the  potential  migration  of  the  current  T&E 
model  to  a  risk-based  model. 
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Recommendations  for  Future  Research 

Actions  that  should  be  taken  are  relatively  straightforward.  These  actions  consist  of 
additional  research  as  well  as  the  initiation  of  a  pilot  program  to  fully  validate  the  concepts 
described  in  this  strategic  research  paper. 

As  referenced  throughout  this  paper,  the  scope  of  additional  research  should  be  expanded 
to  include  a  larger  segment  of  the  program  management  community.  The  next  phase  of  research 
should  include  a  more  robust  target  population  consisting  of  materiel  developers,  combat 
developers,  and  independent  evaluators.  Additionally,  senior  representatives  from  OSD  and  the 
Anny  should  also  be  identified  as  important  stakeholders  in  the  migration  to  risk-based 
evaluations.  A  coordination  forum  should  also  be  convened  to  discuss  major  research  findings, 
develop  specific  processes,  and  implementation  execution  mechanisms. 

Finally,  a  pilot  effort  should  be  planned  to  adequately  characterize  and  correct  any 
inconsistencies  found  in  the  newly  developed  processes.  The  pilot  effort  should  include  small 
sample  hypothesis  tests  performed  on  very  specific  hypothesis  to  formally  test  the  null 
hypothesis.  The  pilot  program  should  consist  of  three  programs  at  varying  stages  of 
development.  A  major  program  of  record  should  be  included  to  accurately  assess  implementation 
risks  and  complexities  for  a  program  approaching  a  production  decision.  A  smaller,  less  complex 
program  approaching  a  Milestone  B  decision  should  be  included  in  the  pilot  to  assess  whether 
the  proposed  risk  management  mechanisms  provide  increased  value.  A  rapid  acquisition  program 
should  be  included  to  characterize  the  variances  associated  with  streamlined  acquisition. 
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ACWA 

GLOSSARY  OF  ACRONYMS  AND  TERMS 

Assembled  Chemical  Weapons  Alternatives 

AEC 

Anny  Evaluation  Center 

AKO 

Anny  Knowledge  Online 

AEP 

Anny  Enterprise  Position 

AMSAA 

Anny  Materiel  Systems  Analysis  Activity 

APG 

Aberdeen  Proving  Ground 

ASA(AL&T) 

Assistant  Secretary  of  the  Anny  for  Acquisition,  Logistics,  and  Technology 

AST 

ATEC  Systems  Team 

ATC 

Aberdeen  Test  Center 

ATEC 

Anny  Test  and  Evaluation  Command 

BMC 

Bridge  Modernization  Command 

CAWG 

Cost  Analysis  Working  Group 

CTMO 

Anny  Civilian  Talent  Management  Office 

CTMP 

Anny  Civilian  Talent  Management  Program 

DA 

Department  of  the  Anny 

DA  PAM 

Department  of  the  Anny  Pamphlet 

DAU 

Defense  Acquisition  University 

DoD 

Department  of  Defense 

DTC 

Developmental  Test  Center 

DUSA 

Deputy  Under  Secretary  of  the  Army 

ECBC 

Edgewood  Chemical  and  Biological  Center 

FY 

Fiscal  Year 

GAO 

General  Accounting  Office  (now  Government  Accountability  Office) 

GO 

General  Officer 

I2A 

Integrated  Intelligence  Architecture 

IED 

Improvised  Explosive  Devices 

IMS 

Integrated  Master  Schedule 

IRL 

Integration  Readiness  Level 

JCIDS 

Joint  Capabilities  Integration  and  Development  System 

JPEO-CBD 

Joint  Program  Executive  Office  for  Chemical  and  Biological  Defense 
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MDA 


Milestone  Decision  Authority 
MRL  Manufacturing  Readiness  Level 

NRC  Nuclear  Regulatory  Commission 

ODASA-CE  Office  of  the  Deputy  Assistant  Secretary  of  the  Army  for  Cost  and 
Economics 

OTC  Operational  Test  Command 

PEO  Program  Executive  Office(r) 

PEO-I  Program  Executive  Office-Integration 

PEO  IEW&S  Program  Executive  Office  for  Intelligence,  Electronic  Warfare,  and  Sensors 

PM  Program  Manager 

PMO  Program  Management  Office 

RDECOM  Research,  Development  and  Engineering  Command 

RSTA  Reconnaissance,  Surveillance,  and  Target  Acquisition 

SAPHIRE  Systems  Analysis  Programs  for  Hands-on  Integrated  Reliability  Evaluations 

SEP  Systems  Engineering  Process 

SES  Senior  Executive  Service 

SSCF  Senior  Service  College  Fellowship  program 

T&E  Test  and  Evaluation 

TRADOC  Training  and  Doctrine  Command  (U.S.  Anny) 

TRL  Technology  Readiness  Level 

USD(AT&L)  Under  Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics 

WIPT  Working  Integrated  Product  Team 
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APPENDIX  A 
SURVEY  INSTRUMENT 


tines  Risk  Based  TAE 


II.  This  survey  inslmment  is  taken  £4  satisfy  the  Strategic  Research  Paper  i-e^uii  enients  oF 
the  D.nLF  Sehi or Service  Co-llege  Fellowship.  My  participation  in  this  research  is  completely 
voluntary,  Do-  we  have  your  consent  to  participate  in  this  survey? 
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2.  What  is  your  assigned  specialty  area? 
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2.  How  many  yeats  of  experience  do  you  have  in  an  acquisition  related  position? 

o  -■IK  till  pill! 
o  -4e  li  lvi  ytan 
I  I  Elixir  k  Him  join 
o  rflnr  I*n-hr 

liir  tw-iiJy  jiira 

4.  What  is  your  current  employment  status? 
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5.  What  is  your  eur-reht  to  hk  gi-a  de  level? 
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£.  Would  you  corisider  yourself  a  producer cr recipient  o-f  Army  Test  and  Evaluation 
products? 
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7.  At  which  major  deeisioh  point;  do  you  fihd  Airny  lest  ard  evaluatioii  products  most 
useful? 
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A.  What  is  the  highest  level  program  van  have  worked  on? 
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Q.  Which  of  the  f  el  lowing  test  and  evaluation  sources  do  von  find  most  useful? 
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IQ.  flow  satisfied  are  yen  with  the  overal  1  quality  oF  the  Following  Ariny  test  and  evaluation 
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12.  Bated  on  your  experience.  which  ri  sk  eoimpaherits  do  you  find  most  important? 
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1  £.  Would  it  be  helpful  if  Army  test  and  evaluation  products  -conducted  cost  based  risk 
analysis  to  the  fol  lowing  areas? 
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IT,  Which  of  the  fallowing  steps  might  benefit  From  risk-based  test  and  evaluation 


products? 
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1 B,  Would  il  be  helpful  if  Army  test  ahd  evaluation  products  conducted  schedule  based 
risk  analysis  to  the  fell  owing  areas? 
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16,  Would  it  be  helpful  if  Army  test  and  evaluation  products  conducted  performance  based 
risk  analysis  to  the  fell  owing  areas? 
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26,  Who  should  be  the  principle  recipient  of  risk-based  test  and  evaluation  products? 
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#21.  Wliat  suggestions,  or  comments  do  you  have  regarding  a  risk  based  approach  to 
Hrhiv  test  arid  evaluation? 
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APPENDIX  B 
NARRATIVE  RESPONSES 
(QUESTION  21) 


What  suggestions  or  comments  do  you  have  regarding  a  risk-based  approach  to  Anny  test 
and  evaluation? 

1.  ATEC  should  move  to  a  risk-based  approach  to  T&E. 

2.  ATEC  should  be  an  active  participant  of  the  risk  management  process  throughout  the 
entire  life  cycle. 

3.  ATEC  should  be  an  active  participant  in  the  PM's  risk  management  effort. 

4.  A  risk-based  approach  would  be  useful  in  characterizing  perfonnance.  Unsure  of  the 
usefulness  in  characterizing  schedule  and  cost  risks. 

5.  Risk-based  testing  would  move  us  from  the  binary,  on-off,  black-white  evaluation  to  a 
more  nuanced  and  useful  construct. 

6.  If  Testers  want  to  be  PMs,  they  should  compete  for  the  job. 

7.  PM’s  are  typically  responsible  to  manage  risk,  not  sure  what  value  T&E  based  risk  adds 
to  PM. 

8.  ATEC  should  assign  risk  to  their  own  products  and  amount  of  time  to  get  their  products 
out  for  usefulness. 

9.  Need  to  know  operational  context  better.  Need  to  keep  pace  with  changing  missions  and 
goals.  Asymmetrical  warfare  and  system  production  to  meet  mission  challenges  change 
requirements,  acceptable  risk,  and  material  solution  dynamics  faster  than  T&E 
community  can  keep  pace. 

10.  T&E  needs  to  customize  its  risk-based  approaches  to  support  source  selection,  post¬ 
award  testing,  and  act  as  an  independent  PARTNER  to  the  PM. 

11.  Risk-based  approach  helps  to  focus  T&E  and  should  be  employed  more  often.  DOT&E  is 
recommending  it. 

12.  Needs  to  be  integrated  earlier  in  the  Acquisition  process  to  have  any  value  added.  As  it 
stands  now,  ATEC  enters  the  process  too  late  to  effectively  impact  design  or  actual  build 
of  the  product.  Operational  testing  is  good  to  see  how  it  works,  but  it's  expensive  and 
hard  to  coordinate  with  technically  immature  systems. 
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13.  It  would  have  been  helpful  if  there  was  more  context  in  understanding  risk-based 
approach  to  Army  test  and  evaluation.  I  could  have  answered  these  questions  differently 
based  upon  situation. 

14.  Risk  analysis  should  be  incorporated  into  every  phase  of  acquisition.  Each  component 
should  ultimately  support  the  PMs  overarching  risk  assessment  and  plan.  I  believe  a 
much  more  focused  training  effort  from  DAU  and  ASA(ALT)  on  risk  would  pay  massive 
dividends. 

15.  Be  punctual.  Late  reports  and  late  information  mean  the  response  is  a  scramble  to 
understand. 

16.  Done  early  when  changes  are  cost-effective. 

17.  More  infonnation  on  what  risk-based  T&E  is  and  how  it  works  should  be  made  available 
to  the  acquisition  community. 

18.  Early  in  the  JCIDS  process,  the  T&E  community  needs  to  be  pulled  into  the  overall,  to 
include  risk  planning,  IMS. 

19.  Risk-based  would  be  better  than  pass/fail. 

20.  The  risk-based  approach  to  T&E  would  be  more  useful  than  the  current  capabilities  & 
limitations  report. 
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APPENDIX  C 

INDEPENDENT  RISK  ASSESSMENT  MODEL 
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APPENDIX  D 

ARMY  TEST  AND  EVALUATION  ORGANIZATION 


Army  Evaluation 
Center 


Operational  Test 
Command 


BG  Regan  I  COL  Richardson 


Legend 
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Independent  Reporting 

T&E  Policy, 

Mandated  by  US  Code, 

Oversight 

OMB,  and  OSD 
< _ 1  _ / 
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APPENDIX  E 

ARMY  TEST  AND  EVALUATION  MISSION  STATEMENT 


ATEC  plans,  integrates,  and  conducts  experiments,  developmental 
operational  testing,  and  independent  evaluations  and  assessments 
information  to  acquisition  decision  makers  and  commanders. 

General  Orders  No.  13 
Signed  CSA  16  Oct  06 


testing,  independent 
to  provide  essential 
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